This disclosure relates generally to the field of image processing. More particularly, but not by way of limitation, it relates to techniques for determining an orientation of an image via the use of image processing techniques, such as face detection and optical character detection (OCD).
In recent years, the proliferation of digital cameras has been rapid, both as stand-alone devices and as part of other consumer products such as, for example, mobile phones and personal digital assistants (PDAs). Many digital cameras are fully or semi-automatic, in the sense that the camera's focus and/or exposure parameters are set automatically (i.e., without direct user input). In addition to setting the focus and/or exposure parameters, some cameras may also automatically determine and store orientation data for the image. In some embodiments, orientation data may be stored in the image's metadata. By storing an image's orientation in metadata, viewing applications and other “offline,” i.e., not executing on the image capture device itself, photo editing tools may be able to automatically display the image to the user in the orientation that the image was captured in.
As used herein, an image's orientation may be determined in relation to the plane of the image capturing device's camera sensor array (i.e., image sensor) with respect to the gravity vector, g. Referring to FIG. 1, “Portrait Up” may be defined as that device orientation 100a in which the side of the device spanning the upper left corner of the device (UL) and the upper right corner of the device (UR), i.e., the “UL-UR side,” is above the side of the device spanning the lower left corner of the device (LL) and the lower right corner of the device (LR), i.e., the “LL-LR side,” in relation to gravity vector 102. Similarly, “Portrait Down” may be defined as that device orientation 100b in which the side of the device spanning the lower right corner of the device (LR) and the lower left corner of the device (LL), i.e., the “LR-LL side,” is above the side of the device spanning the upper right corner of the device (UR) and the upper left corner of the device (UL), i.e., the “UR-UL side,” in relation to gravity vector 102. Likewise, “Landscape Left” may be defined as that device orientation 100c in which the “UR-LR side” is above the “UL-LL side,” and “Landscape Right” may be defined as that device orientation 100d in which the “LL-UL side” is above the “LR-UR side.” For device positions in between the four absolute device orientation positions depicted in FIG. 1 (i.e., “Portrait Up,” “Portrait Down,” “Landscape Left,” and “Landscape Right”), the image processing algorithms may choose the “most-likely intended orientation” using the detailed positional sensor data returned by the device's positional sensors, E.g., if the device is tilted less than 45 degrees to the left or right from the “Portrait Up” position, the image processing algorithm may determine that “Portrait Up” is still the most likely image orientation.
Device positional sensors may comprise an accelerometer, e.g., a microelectromechanical systems (MEMS) accelerometer, such as an ultra compact low-power three-axes linear accelerometer. An accelerometer may include a sensing element and an integrated circuit (IC) interface able to provide the measured acceleration of the device through a serial interface. Acceleration measurements reported by the accelerometer may take the form of electrical measurements indicative of the relative pull of gravity on the device along its three primary axes.
Turning now to FIG. 2, various primary axes 104 of an electronic image capture device 100e in relation to a gravity vector 102 are shown. As shown in FIG. 2, the x-axis pierces the longer edge of the display screen, with positive x values to the right side (as shown in FIG. 2) and negative values to the left side. Likewise, the y-axis pierces the shorter edge of the display screen, with positive y values to the top side (as shown in FIG. 2) and negative values to the bottom side. As is displayed more clearly in the context of the side view of image capture device 100f in FIG. 2, the z-axis pierces through the display screen of the device, with positive z values to the side of the device having the display and negative values to the back side of the device.
As oriented in FIG. 2, i.e., in the “Portrait Up” orientation, an exemplary accelerometer may return positional data indicating that 100% of the force of gravity is acting in the negative y-axis direction of image capture device 100. Likewise, if the image capture device were instead in the “Portrait Down” orientation, the exemplary accelerometer may return positional data indicating that 100% of the force of gravity is acting in the positive y-axis direction. No matter which direction the image capture device is oriented, the direction of gravitational pull remains towards the center of the Earth.
In some scenarios, device positional sensor information may prove unreliable in determining the orientation of the device with respect to a captured image. Consider the scenario where the image capture device is lying flat on a table top or other horizontal surface. As can be seen from FIG. 2, the force of gravity would be pulling exclusively in the direction of the z-axis if the image capture device were lying on a flat horizontal surface. Thus, if a user rotated the phone while it was lying on the flat horizontal surface, image processing routines would be unable to discern whether the image was being held in portrait or landscape orientation via the accelerometer information. This is because the pull of gravity along the x- and y-axes would remain negligible so long as the device was lying flat, no matter how the device was rotated.
Turning now to FIG. 3, an exemplary scenario is shown wherein image orientation may not be reliably determined by the positional sensors of an electronic capture device 100g. In FIG. 3, the image capture device 100g is being held nearly parallel to a sheet of paper 306 with written text 308 on it that is lying on a flat surface. In other words, the gravity vector 102, g, is acting almost entirely in the negative z-axis direction. Thus, rotation 310 of the image capture device around its z-axis will not be perceived by the device's accelerometer as a change in orientation for the reasons discussed above. Dashed line 300 represents the normal plane to the gravity vector, g, and angle 304 is indicative of the angular offset of the image capture device from plane 300, i.e., the level position. In some embodiments, if angle 304 is determined to be within plus or minus, say, 15 degrees from plane 300, the image capture device is deemed to be positioned in a scenario wherein the positional information data is unreliable for determining device orientation. When the device is positioned greater than 15 degrees above or below plane 300, enough of the gravity vector, g, is acting along the device's x- and y-axes that a reliable determination of image orientation may be determined from the device's positional sensor data. As mentioned above, in some exemplary scenarios having unreliable position sensor data, there may be text 308 that is captured by the device's camera sensor array and displayed on the device's display in the form of text 312. It would be helpful if a determination of image orientation in such unreliable position sensor scenarios could be made using image processing techniques, e.g., technique involving OCD metrics.
Accordingly, there is a need for techniques to improve image orientation detection in scenarios in which device positional sensor information proves unreliable. By employing image processing techniques to image orientation detection, more effective image orientation determinations may be made in scenarios during which the image capturing device is in a position where an orientation cannot reliably be determined from positional sensor information returned from the device. By using novel, rotationally invariant character detection techniques, optionally combined with face detection techniques, a robust image orientation determination system may be employed, even in situations where reliably determining image orientation was previously thought to be impossible.